<!DOCTYPE html>
<html lang="en">
<head>
        <title>Stratix V accelerator card from eBay</title>
        <meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link rel="shortcut icon" href="./theme/images/favicon.ico"/>
        <link rel="stylesheet" href="./theme/css/main.css" type="text/css" />
        <link href="www.j-marjanovic.io/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="j-marjanovic.io Atom Feed" />

        <!--[if IE]>
                <script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script><![endif]-->

        <!--[if lte IE 7]>
                <link rel="stylesheet" type="text/css" media="all" href="./css/ie.css"/>
                <script src="./js/IE8.js" type="text/javascript"></script><![endif]-->

        <!--[if lt IE 7]>
                <link rel="stylesheet" type="text/css" media="all" href="./css/ie6.css"/><![endif]-->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8/jquery.min.js" type="text/javascript"></script>
<script type="text/javascript" src="theme/jquery.cookiesdirective.js"></script>

</head>

<body id="index" class="home">
<script type="text/javascript">
	// Using $(document).ready never hurts
	$(document).ready(function(){

		// Cookie setting script wrapper
		var cookieScripts = function () {
			// Internal javascript called
			console.log("Running");
		
			// Loading external javascript file
			$.cookiesDirective.loadScript({
				uri:'external.js',
				appendTo: 'eantics'
			});
		}
	
		/* Call cookiesDirective, overriding any default params
		
			*** These are the defaults ***
				explicitConsent: true,
				position: 'top',
				duration: 10,
				limit: 0,
				message: null,				
				cookieScripts: null,
				privacyPolicyUri: 'privacy.html',
				scriptWrapper: function(){},	
				fontFamily: 'helvetica',
				fontColor: '#FFFFFF',
				fontSize: '13px',
				backgroundColor: '#000000',
				backgroundOpacity: '80',
				linkColor: '#CA0000'
				
		*/
		
		$.cookiesDirective({
			privacyPolicyUri: 'myprivacypolicy.html',
			explicitConsent: false,
			position : 'bottom',
			scriptWrapper: cookieScripts, 
			cookieScripts: 'Google Analytics', 
			backgroundColor: '#52B54A',
			linkColor: '#ffffff'
		});
	});
</script>

	
  <!-- <header id="banner" class="body"> -->
  <!--               <h1><a href="./"><img src="http://www.launchyard.com/images/logo.png" /></a></h1> -->
  <!--       </header> --> 

  <div class="LaunchyardDetail" style="align:right;">
    <!-- <p> -->
    <!-- <img src="./theme/images/blue-pin.png" width="100" height="100" alt="Graph icon"> -->
    <!-- </p> -->
    <p><a id="sitesubtitle" href="./">j-marjanovic.io</a></p>

	<br>
    <p style="float: right; margin-right: 50px;"><a id="aboutlink" href="./pages/about.html">About</a></p>

    <br>
	<p style="float: right; margin-right: 50px;"><img src="./theme/images/icons/rss.png"> <a id="aboutlink" href="./feeds/jan-marjanovic.atom.xml">Atom feed</a></p>
    <br>

  </div>

<section id="content" >
    <div class="body">
      <article>
        <header>
          <h1 class="entry-title">
            <a href="./stratix-v-accelerator-card-from-ebay.html" rel="bookmark"
               title="Permalink to Stratix V accelerator card from eBay">Stratix V accelerator card from eBay</a></h1>

        </header>

        <div class="entry-content">
<div class="post-info">
	<ul>
        <li class="vcard author">
                 by&nbsp;<a class="url fn" href="./author/jan-marjanovic.html">Jan Marjanovic</a>
        </li>
        <li class="published" title="2020-05-03T14:30:00+02:00">
          on&nbsp;Sun 03 May 2020
        </li>

	</ul>

</div><!-- /.post-info -->          <p>A couple of weeks ago <a href="https://twitter.com/rombik_su">@rombik_su</a> in a <a href="https://twitter.com/rombik_su/status/1250382904074608642">Twitter
thread</a> pointed out a
very cheap FPGA accelerator card on eBay. The board contains a proprietary
Samtec board-to-board connector, most likely carrying power, PCI Express, and
auxiliary signals (JTAG, IPMI to BMC, ...), two QSFP cages, DDR3 and a large
FPGA, hidden under the heatsink.</p>
<p>Being passionate about everything FPGA-related, and with Coronavirus lockdown
limiting the number of fun things to do, I decided to purchase the board. An
evaluation kit of this kind can easily cost thousands, and 40 USD is a real
bargain.</p>
<h2>Initial research</h2>
<p>While waiting for the board to arrive, I did some initial investigation. The
description on eBay is quite cryptic. The title of the listing included all the
text from the labels on the board, including the label "AIRFLOW" indicating
the direction of the forced air through the board.</p>
<p>One of the most fruitful clues was the label "Microsoft" on the board. It was
well-publicized a couple of years ago that Microsoft is using FPGAs to
accelerate Bing searches, and this might be one of the boards used in the
servers.</p>
<p>Remembering that Microsoft went with Altera (now Intel PSG) and that this was
some years back, it is most likely that the card contains a Stratix V FPGA.</p>
<p>The <a href="https://www.microsoft.com/en-us/research/uploads/prod/2014/06/HC26.12.520-Recon-Fabric-Pulnam-Microsoft-Catapult.pdf">first
link</a>
on Google (or should I have used Bing? would the FPGAs be aware that I am
looking for information about them?) for "microsoft catapult stratix v"
presented some conceptually similar cards, but not exactly the same.</p>
<p>The <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/10/Cloud-Scale-Acceleration-Architecture.pdf">second
link</a>,
however, presented the exact card which I have purchased:</p>
<p><img alt="FPGA card, from A. Caulfield et al: A Cloud-Scale Acceleration Architecture" src="./images/2020_fpga_card/img-002.jpg" style="width:60%; display: block; margin-left: auto; margin-right: auto;"></p>
<div style="text-align: center;"><small>(from A. Caulfield et al: A Cloud-Scale Acceleration Architecture)</small></div>

<p>&nbsp;</p>
<p>The paper also contains a block diagram highlighting the main components
of the board:</p>
<p><img alt="Block diagram, from A. Caulfield et al: A Cloud-Scale Acceleration Architecture" src="./images/2020_fpga_card/block_diagram.png" style="width:60%; display: block; margin-left: auto; margin-right: auto;"></p>
<div style="text-align: center;"><small>(from A. Caulfield et al: A Cloud-Scale Acceleration Architecture)</small></div>

<p>&nbsp;</p>
<p>The paper also mentions "widely-used OpenCompute server" which could give out
some information about the pinout on the Samtec connector. Since there are
numerous formats for mezzanine cards standardized by the OpenCompute project,
and finding the matching one is not trivial, I decided to leave this task for
later.</p>
<p>Further digging in the search results, I was also able to find <a href="https://mspoweruser.com/microsoft-talks-about-the-project-that-helped-them-build-the-worlds-first-ai-supercomputer/">another photo</a>
of the board with the heatsink removed.</p>
<p>The marking on the FPGA are removed, but we already know that it is a Stratix V,
and some things were expected from previous documents, e.g. 5 DDR3 chips for
a combined of 9 chips for 72-bit DDR3 data width).</p>
<p>What I found interesting in this picture is a Flash memory (Micron 25Q256) in
the top right corner, which means that the image for the FPGA is most likely
stored on the board and is not downloaded through the connector at the startup.
It is also possible that <a href="https://www.intel.com/content/www/us/en/programmable/support/support-resources/support-centers/devices/cfg-index/cfg-via-protocol.html">Configuration via
Protocol</a>
is used, and only the basic image is stored in the Flash.</p>
<h1>Board overview</h1>
<p>Then one day, the board finally arrives. The board matches the description
in the previously-mentioned article.</p>
<p>A quick look at the board identifies the following components:</p>
<ul>
<li>160 pin Samtec SEARAY™ connector</li>
<li>USB (FT232H, used as a JTAG interface)</li>
<li>power entry (FDMS0310AS MOSFET, 0.01 Ohm resistor, unidentified IC with the markings "L536FCD")</li>
<li>power converters (Enpirion® EN2342QI DC-DC converter)</li>
<li>programmable oscillator (IDT8N4Q001)</li>
<li>Flash memory for FPGA (Micron N25Q256A)</li>
<li>I2C EEPROM (ST M24128-BW)</li>
<li>DDR3 memory (SK hynix H5TC4G83BFR)</li>
</ul>
<p style="width:80%; display: block; margin-left: auto; margin-right: auto;"><img alt="The board with annotated sections" src="./images/2020_fpga_card/9560_annotated.jpg"></p>
<h1>USB/JTAG</h1>
<p>At this point, it was time to start experimenting with the board. The FT232H
seems to be powered from the USB BUS voltage: pin 1 of J3 is connected to pin 40
(VREGIN) of FT232H. This is why I decided to first start with the USB. With the
datasheet for FT232H it was trivial to determine the pin assignments on the
connector. I did not want to solder directly on the connector pins, as I would
like to make a proper cable in the future.</p>
<p style="width:80%; display: block; margin-left: auto; margin-right: auto;"><img alt="USB connection with signal annotations" src="./images/2020_fpga_card/9570_annotated.jpg"></p>
<p>After plugging the cable in the computer, the FT232H is recognized as "USB
Serial Converter":</p>
<p style="width:40%; display: block; margin-left: auto; margin-right: auto;"><img alt="Driver information about the FT232H" src="./images/2020_fpga_card/device_manager2.png"></p>
<h1>Connector, 1st look</h1>
<p>At this point I could start trying to reverse engineer the connector pinout. I
managed to figure out the following connections:</p>
<ul>
<li>Ground</li>
<li>Input power: there are a couple of pins connected to the Drain of the large
    MOSFET. I also assume that the level is, as it is common for PCIe cards,
    12V.</li>
<li>There are 16 pairs of AC-coupling capacitors near the connector. PCIe
    standard mandates capacitors on the TX side, so I assume this is PCIE_TX.
    Not knowing the exact lane numbering, I decided to enumerate them with
    letters instead of numbers.</li>
<li>PCIe RX is TBD, but looking at unassigned pins a clear pattern is visible</li>
<li>Some of the pins are connected to the circuit above the connector - I have
    annotated these pins, but right now I do not have a clear idea what is the
    purpose. I annotated those pins as AUX.</li>
</ul>
<p style="width:100%; display: block; margin-left: auto; margin-right: auto;"><img alt="Samtec connector with partial pinout, 1" src="./images/2020_fpga_card/pinout_1.png"></p>
<h1>Power</h1>
<p>I have connected the 12V on the Drain side of the Q12 (FDMS0310AS), which has
also energized the PU12 and +12V pins on the Samtec connector. However, the
power consumption was only 3 mA, and the MOSFET was left closed.</p>
<p>I could not find any information about PU12 (L536FCD). I assume it is some
kind of a current-limit protection, measuring the current through the 0.01 Ohm
shunt resistor and controlling the Gate pin of Q12.</p>
<p>To literally bypass this problem, I have decided to also connect 12 V on the
other side of the Q12. This yielded some results; the power consumption raised
to 695 mA, which is what one would expect from such board, and the LEDs turned
on. </p>
<p style="width:80%; display: block; margin-left: auto; margin-right: auto;"><img alt="Power consumption" src="./images/2020_fpga_card/fr_9586_size1024.jpg"></p>
<p>On various points in the circuit I could also measure all the voltages one
would expect to find in such a circuit:</p>
<ul>
<li>1.5V on the Enpirion (PU16) output</li>
<li>1.35 V on C406 (DDR3 voltage)</li>
<li>0.674 V on C1005 (DDR3 termination voltage)</li>
<li>2.5 V on C442 (periphery)</li>
<li>3.3 V on QSFP capacitors</li>
<li>0.9 V on C354 (FPGA core voltage)</li>
</ul>
<h1>Memories</h1>
<h2>FPGA configuration memory</h2>
<p>The heatsink covers the large part of the top side of the board including the
Flash memory for the FPGA configuration. One can, however, still reach the pins
8 and 9 (DQ1 and DQ2, respectively) with an oscilloscope probe.</p>
<p>After the power is applied to the board, we can observe that the FPGA gets
programmed in roughly a second. This is above the 100 ms/200 ms limit required
by the PCIe standard, but in this custom form factor the value might be
different.</p>
<p>DQ2 pin on N25Q256A:</p>
<p style="width:80%; display: block; margin-left: auto; margin-right: auto;"><img alt="Data from DQ2 pin after power up" src="./images/2020_fpga_card/mem_top_left_pin.png"></p>
<p>LEDs on the board are also driven by the FPGA and remain lit for a second until
the FPGA is not programmed, another indication that the FPGA gets configured
from the memory.</p>
<h2>EEPROM</h2>
<p>Another memory on the board is a small 128Kbit EEPROM, which probably stores MAC
addresses, serial numbers, and other similar information.</p>
<p>Quite interestingly, SDA and SCL lines remain stuck low after some time.
Maybe the EEPROM is not used at all, or maybe there is some other part
of the circuit is keeping the EEPROM interface state machine in a reset.</p>
<p>SDA pin on M24128-BW, SCL is very similar:</p>
<p style="width:80%; display: block; margin-left: auto; margin-right: auto;"><img alt="SDA pin on M24128-BW" src="./images/2020_fpga_card/M24128_SDA.png"></p>
<h1>Connector, 2nd look</h1>
<p>With the board powered on, I could measure the voltage on the connector pins.
As expected, on 16 differential pairs I can sense a bias voltage of the PCIe
receivers, around 0.7 V.</p>
<p>Some of the AUX pins have a slight bias, but this is probably caused by
pull-up resistors and other components. It would require more investigation
to fully understand the purpose of these pins.</p>
<p style="width:100%; display: block; margin-left: auto; margin-right: auto;"><img alt="Samtec connector with partial pinout, 2" src="./images/2020_fpga_card/pinout_2.png"></p>
<h1>Components</h1>
<h2>QSFP</h2>
<p>Plugging in a QSFP cable only marginally increases the power consumption.
It is clear that the high-speed circuit is disabled, most likely because
of an internal register configuration and less likely because the cable
is "not compatible", i.e. the board parses the EEPROM and it does not enable
the high-speed circuit.</p>
<p>Consumption from the 12V input:</p>
<ul>
<li>without the QSFP module inserted: 696.7 mA</li>
<li>with the QSFP module inserted: 697.5 mA</li>
</ul>
<h2>IDT oscillator</h2>
<p>IDT oscillator is producing 645 MHz clock.</p>
<h2>DDR3</h2>
<p>I have probed what I presume are the DDR3 termination resistors (on the bottom
side, on the other side of the 5th DDR3 component on the top) and observed
no switching activity. It seems that the DDR3 controller is kept in reset.</p>
<h1>Outlook</h1>
<p>The first bring-up session was quite successful; I have managed to turn on the
board without damaging it and figure out the basic pinout of the connector.</p>
<p>For the next step, I plan to investigate the OpenCompute website if there exists
a document that would describe the pinout of the connector. Still left to be
determined are the reference clock and an enable signal from the connector.</p>
<p>Eventually I plan to produce a small PCB that would allow plugging this board
in a normal PCIe card slot.</p>
<hr>
<p>All trademarks and registered trademarks are the property of their respective owners.</p>
        </div><!-- /.entry-content -->
<a href="https://twitter.com/share" class="twitter-share-button" data-count="horizontal" data-via="janmarjanovic">Tweet</a><script type="text/javascript" src="https://platform.twitter.com/widgets.js"></script><br/><br/>

      </article>
    </div>
</section>
        <section id="extras" >
        
        </section><!-- /#extras -->
	
        <footer id="contentinfo" >
                <address id="about" class="vcard ">
                Proudly powered by <a href="https://getpelican.com/" target="_blank">Pelican</a>, which takes
                great advantage of <a href="https://python.org" target="_blank">Python</a>.
		
                </address><!-- /#about -->
		

                
        </footer><!-- /#contentinfo -->

<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-56555055-1', 'auto');
  ga('send', 'pageview');

</script></body>
</html>